Image processing apparatus, image processing system, image processing method, and recording medium

ABSTRACT

An image processing apparatus including a storage unit, a correction unit, and an output unit. The storage unit stores moving-image data including a plurality of frames captured by an image-capturing device communicable with the image processing apparatus, time-series data of an inclination angle with reference to a reference direction of the image-capturing device, and time-series data of an angle velocity of the image-capturing device. The correction unit, based on the time-series data of the angle velocity, rotates an image of each of the plurality of frames of the moving-image data to reduce a rotational distortion around the reference direction within a prescribed frequency range. The output unit outputs image data of the rotated image of each of the plurality of frames to an external device communicable with the image processing apparatus.

TECHNICAL FIELD

The embodiments of the present disclosure relate to an image processingapparatus, an image processing system, image processing method, andrecording medium.

BACKGROUND ART

In recent years, hand-held cameras capable of capturing still images ormoving images with a hemispherical or omnidirectional field of view arewidely used. With such hand-held cameras, it is difficult to accuratelycontrol the position (angle) of the camera when shooting. In view ofsuch a situation, there are known cameras capable of correcting theinclination of an image captured with the camera body inclined.

The technology for correcting the inclination of an image is disclosed,for example, in JP-2013-214947-A. JP-2013-214947-A discloses an imagecapturing apparatus having two fish-eye lenses that calculates atransformation parameter for transforming a fish-eye image into aholomorphic image, in accordance with an inclination angle detected byan acceleration sensor disposed inside the image capturing apparatus fordetecting an inclination angle of the image capturing apparatus withreference to the vertical direction. However, the correction devicedisclosed in JP-2013-214947-A fails to correct the rotational distortionof the image around the vertical direction, which is because only thecorrection for the inclination is made using the acceleration sensor inthe apparatus of JP-2013-214947-A.

Further, JP-2017-147682-A discloses an omnidirectional imaging systemincluding an image capturing unit, an image synthesizing unit thatcreates omnidirectional frame data, a posture detecting unit thatdetects posture information of the image capturing unit, acorrection-amount calculating unit that generates correction-amount datafor transforming the coordinates of the omnidirectional frame data basedon the posture information, and an associating unit that associates theomnidirectional frame data with the correction-amount data to obtain anomnidirectional image. In the omnidirectional imaging system ofJP-2017-147682-A, the correction-amount calculating unit generates, forthe omnidirectional frame data, the correction-amount data used tocorrect the inclination with reference to the vertical direction in theglobal coordinate and to remove a small vibration component from thehorizontal plane vibration, thus to transmit the correction-amount datato the associating unit.

However, the technology disclosed in JP-2017-147682-A adverselyincreases the load during the shooting because the posture informationof the image capturing unit is detected and the correction-amount datais calculated during the shooting. Further, since only the correctionamount data used to remove the small vibration component within thehorizontal plane in the global coordinate system is stored inassociation with the omnidirectional frame data, the informationregarding the small vibration component is unavailable in the system ofJP-2017-147682-A.

CITATION LIST Patent Literature

[PTL 1] JP-2013-214947-A

[PTL 2] JP-2017-147682-A

SUMMARY OF INVENTION Technical Problem

It is difficult to capture an image with the stationary position andposture while holding the camera with a hand. If images are capturedwith the camera body inclined, the zeniths of the images are misalignedand the horizontal line is distorted unless the zenith correction isperformed. If the camera body rotates due to, for example, camera shake,the image blurs along the horizontal line unless the rotation correctionis performed.

Solution to Problem

In view of the above, there is provided image processing apparatusincluding a storage unit, a correction unit, and an output unit.Advantageously, the storage unit stores moving-image data including aplurality of frames captured by an image-capturing device communicablewith the image processing apparatus, time-series data of an inclinationangle with reference to a reference direction of the image-capturingdevice, and time-series data of an angle velocity of the image-capturingdevice. The correction unit, based on the time-series data of the anglevelocity, rotates an image of each of the plurality of frames of themoving-image data to reduce a rotational distortion around the referencedirection within a prescribed frequency range. The output unit outputsimage data of the rotated image of each of the plurality of frames to anexternal device communicable with the image processing apparatus.

Advantageous Effects of Invention

Accordingly, the rotational blur of the image around the prescribedreference direction can be effectively corrected.

BRIEF DESCRIPTION OF DRAWINGS

The aforementioned and other aspects, features, and advantages of thepresent disclosure will be better understood by reference to thefollowing detailed description when considered in connection with theaccompanying drawings. The accompanying drawings are intended to depictembodiments of the present disclosure and should not be interpreted tolimit the scope thereof. The accompanying drawings are not to beconsidered as drawn to scale unless explicitly noted.

FIG. 1 is a cross-sectional view of an omnidirectional cameraconstituting an omnidirectional moving-image system according to anembodiment of the present disclosure.

FIGS. 2A and 2B (FIG. 2) are block diagrams of a hardware configurationdiagram of the omnidirectional moving-image system according to anembodiment of the present disclosure.

FIG. 3 is a block diagram of a functional configuration of anomnidirectional moving-image correction function implemented by theomnidirectional moving-image system according to an embodiment of thepresent disclosure.

FIG. 4A is a data flow diagram of the omnidirectional moving-imagesystem according to an embodiment of the present disclosure.

FIG. 4B is an illustration of a plane data configuration of anomnidirectional image generated by the omnidirectional moving-imagesystem according to an embodiment of the present disclosure.

FIG. 4C is an illustration of a spherical data configuration of theomnidirectional image generated by the omnidirectional moving-imagesystem according to an embodiment of the present disclosure.

FIG. 5 is an illustration of zenith correction and rotation correctionof an omnidirectional image in the omnidirectional moving-image systemaccording to an embodiment of the present disclosure.

FIGS. 6A and 6B (FIG. 6) are illustrations of omnidirectional imagesobtained by performing the zenith correction and the rotation correctionaccording to an embodiment of the present disclosure.

FIG. 7 is a flowchart of an omnidirectional moving-image correctionprocessing performed by the omnidirectional camera in FIG. 1.

FIGS. 8A and 8B (FIG. 8) are illustrations of a method of calculating afront vector used for the rotation correction according to an embodimentof the present disclosure.

DESCRIPTION OF EMBODIMENTS

In describing preferred embodiments illustrated in the drawings,specific terminology is employed for the sake of clarity. However, thedisclosure of this patent specification is not intended to be limited tothe specific terminology so selected, and it is to be understood thateach specific element includes all technical equivalents that have thesame function, operate in a similar manner, and achieve a similarresult. In the following embodiments, an image processing apparatus andan image processing system are described using an omnidirectional cameraand an omnidirectional moving-image system. Hereinafter, the overallconfiguration of an omnidirectional moving-image system 1000 accordingto an embodiment of the present disclosure will be described withreference to FIGS. 1 and 2.

FIG. 1 is a cross-sectional view of an omnidirectional camera 110 thatconstitutes the omnidirectional moving-image system 1000 according to anembodiment of the present disclosure. The omnidirectional camera 110 inFIG. 1 includes an imaging body 12, a housing 14 that holds the imagingbody 12 and components such as a controller (a central processing unit(CPU 112)) and a battery, and a shutter button 18 provided on thehousing 14. The imaging body 12 in FIG. 1 includes two image-formingoptical systems 20A and 20B and two image sensors 22A and 22B. Examplesof the image sensors 22A and 22B include charge-coupled devices (CCDs)and complementary metal oxide semiconductors (CMOSs). The image-formingoptical systems 20A and 20B are hereinafter sometimes referred tocollectively as an image-forming optical system 20. The image sensors22A and 22B are hereinafter sometimes referred to collectively as animage sensor 22. Each of the image-forming optical systems 20A and 20Bis configured as a fish-eye lens consisting of, for example, sevenlenses in six groups. In the embodiment illustrated in FIG. 1, theabove-mentioned fish-eye lens has a full angle of view of greater than180 degrees (=360 degrees/n where n denotes the number of opticalsystems and n is 2). Preferably, the fish-eye lens in FIG. 1 has anangle of view of 190 degrees or greater. One of such wide-angleimage-forming optical systems 20 (20A and 20B) is combined withcorresponding one of the image sensors 22 (22A and 22B) to constitute awide-angle imaging optical system (20 and 22).

The relative positions of the optical elements (lenses, prisms, filters,and aperture stops) of the two image-forming optical systems 20A and 20Bare defined with reference to the image sensors 22A and 22B. Morespecifically, these elements are positioned such that the optical axisof the optical element of each of the image-forming optical systems 20Aand 20B meets the central portion of the light receiving area ofcorresponding one of the image sensors 22 at the right angle and suchthat the light receiving area serves as the image-forming plane ofcorresponding one of the fish-eye lenses.

In the embodiment illustrated in FIG. 1, the image forming opticalsystems 20A and 20B have the same specification and are combined indirections reverse to each other such that the optical axes thereofmatch with each other. The image sensors 22A and 22B transform the lightdistribution of the received light into image signals, and sequentiallyoutput image frames to an image processing unit (an image processingblock 116) of the controller (the CPU 112). As will be described laterin detail, the images captured by the respective image sensors 22A and22B are combined to generate an image over a solid angle of 4 asteradian (hereinafter, such an image is referred to as a “sphericalimage”). The omnidirectional image is an image of all the directionsthat can be seen from an image capturing point. Thus-obtainedconsecutive frames of the omnidirectional image form an omnidirectionalmoving image. In the following embodiments, cases where anomnidirectional image and omnidirectional moving image are generated aredescribed. In some embodiments, a full-circle image and a full-circlemoving image, a panoramic image and a panoramic moving image obtainedmay be generated. Note that such a panoramic image and moving image areobtained by photographing 360 degrees only in a horizontal plane.

FIG. 2A is an illustration of a hardware configuration of theomnidirectional camera 110 that constitutes the omnidirectionalmoving-image system 1000 according to an embodiment of the presentdisclosure. The omnidirectional camera 110 corresponds to an imageprocessing apparatus or an image-capturing device according to anembodiment to be described.

The omnidirectional camera 110 includes the CPU 112 (a first CPU), aread only memory (ROM) 114, an image processing block 116, amoving-image compressing block 118, a dynamic random access memory(DRAM) 132 that is connected thereto through a DRAM interface 120, and asensor 136 that is connected thereto through a sensor interface 124.

The CPU 112 controls the operations of components of the omnidirectionalcamera 110. The ROM 114 stores therein a control program described in acode readable by the CPU 112 and various kinds of parameters. The imageprocessing block 116 is connected to a first image sensor 130A and asecond image sensor 130B (corresponding to the image sensors 22A and 22Bin FIG. 1, respectively), and receives image signals of images capturedby the image sensors 130A and 130B. The image processing block 116includes, for example, an image signal processor (ISP), and applies, forexample, shading correction, Bayer interpolation, white balancecorrection, and gamma correction to the image signals received from theimage sensors 130A and 130B.

The moving-image compressing block 118 is a codec block for compressingand expanding a video such as video in MPEG-4 AVC/H.264 format. The DRAM132 provides a storage area for temporarily storing data therein whenvarious types of signal processing and image processing are applied.

The sensor 136 is a sensor for detecting three-axis accelerationcomponents and three-axis angular velocity components. The detectedacceleration component and angular velocity component are used toperform zenith correction of the omnidirectional image in the directionof gravity and rotation correction around the direction of gravity asdescribed later. The sensor 136 may further include a sensor such as ageomagnetic sensor for obtaining, for example, an azimuth angle.

The omnidirectional camera 110 further includes a storage interface 122,a universal serial bus (USB) interface 126, and a serial block 128. Thestorage interface 122 is connected to an external storage 134. Thestorage interface 122 controls reading and writing to the externalstorage 134, such as a memory card inserted in a memory card slot. TheUSB interface 126 is connected to a USB connector 138. The USB interface126 controls USB communication with an external device such as asmartphone via the USB connector 138. The serial block 128 controlsserial communication with an external device such as a smartphone and isconnected to a wireless network interface card (NIC) 140.

When the power is turned on by the operation of a power switch, thecontrol program mentioned above is loaded into the main memory (the DRAM132). The CPU 112 controls the operations of the parts of the device andtemporarily stores the data required for the control in the memoryaccording to the program read into the main memory. This operationimplements functional units and processes of the omnidirectional camera110, as will be described later.

FIG. 2B is an illustration of a hardware configuration of theinformation terminal 150 of the omnidirectional moving-image system 1000according to the present embodiment. The information terminal 150 is anexample of an information processing apparatus according to anembodiment of the present disclosure.

The information terminal 150 in FIG. 2B includes a CPU 152, a RAM 154,an internal storage 156, an input device 158, a storage 160, a display162, a wireless NIC 164, and a USB connector 166.

The CPU 152 controls the operations of each and entire component of theinformation terminal 150. The RAM 154 provides the work area of the CPU152. The internal storage 156 stores a control program such as anoperating system described with code that can be decrypted by the CPU152 and an application that processes the information terminal 150according to the present embodiment.

The input device 158 is an input device such as a touch screen andprovides a user interface. The input device 158 accepts variousinstructions from the operator, for example, to correct theomnidirectional moving image. The storage 160 is a removable recordingmedium inserted, for example, into a memory card slot of the informationterminal 150, and records various types of data, such as image data in avideo format and still image data. The display 162 displays thecorrected omnidirectional moving image on the screen in response to theuser operation. The wireless NIC 164 provides a wireless communicationconnection with an external device such as the omnidirectional camera110. The USB connector 166 provides a USB connection to an externaldevice such as the omnidirectional camera 110.

When the information terminal 150 is powered on and the power supply isturned on, the control program is read from the internal storage 156 andloaded into the RAM 154. The CPU 152 control the operation of each partof the apparatus and temporarily stores the data required for thecontrol in the memory, according to the control program read into theRAM 154. This operation implements functional units and processes of theinformation terminal 150, as will be described later.

Hereinafter, the omnidirectional moving-image correction function of theomnidirectional moving-image system 1000 according to the presentembodiment are described with reference to FIGS. 3 to 6.

FIG. 3 is an illustration of functional blocks related to anomnidirectional moving image correction function implemented on theomnidirectional moving-image system 1000 according to the presentembodiment of the present disclosure. As illustrated in FIG. 3, thefunctional block 210 of the omnidirectional camera 110 includes anomnidirectional moving-image capturing unit 212, a storage unit 214, anomnidirectional moving-image correction unit 220 as a correction unit, areceiving unit 230, and an image output unit 232 as an output unit.Further, a functional block 250 of the information terminal 150 includesan instruction receiving unit 252 and a display unit 254.

The functional block 210 of the omnidirectional camera 110 is firstdescribed. The omnidirectional moving-image capturing unit 212sequentially captures consecutive frames using the two image sensors130A and 130B, and records the omnidirectional moving-image data 240 inthe storage unit 214. Further, the omnidirectional moving-imagecapturing unit 212 measures acceleration components in the three axialdirections and angular velocity components in the three axial directionsusing the sensor 136, and records the measured information as themetadata of the omnidirectional moving-image data 240.

The storage unit 214 stores one or more pieces of moving-image data 240recorded by the omnidirectional moving-image capturing unit 212. Thestorage unit 214 serves as a part of a storage area of the externalstorage 134 or another storage in FIG. 2. As illustrated in FIG. 3, theomnidirectional moving-image data 240 includes omnidirectional framedata 242 as moving-image data captured by the omnidirectional camera110, zenith correction data 244 that is time-series data of theinclination angle of the omnidirectional camera 110 with respect to thereference direction of the omnidirectional camera 110 during shooting,and angle velocity data 246 that is time-series data of the anglevelocity of the omnidirectional camera 110 during the shootingoperation.

The omnidirectional frame data 242 includes a plurality of framesconstituting each omnidirectional moving image during the time from thestart to the end of an imaging operation. That is, the omnidirectionalframe data 242 is moving-image data that constitutes the omnidirectionalmoving-image data 240.

Hereinafter, a description will be given of how to generate anomnidirectional image and the generated omnidirectional image withreference to FIGS. 4A, 4B, and 4C. FIG. 4A is an illustration of datastructure of each image and the data flow of the image in generating anomnidirectional image. First, an image directly captured by each of theimage sensors 130A and 130B is an image that roughly covers a hemisphereof the whole sphere as a field of view. Light that passes through theimage-forming optical system 20 is focused on the light receiving areaof the image sensor 130 to form an image according to a predeterminedprojection system. The image sensor 130 is a two-dimensional imagesensor defining a planar area of the light receiving area. Accordingly,the image formed by the image sensor 130 is image data represented by aplane coordinate system. Such a formed image is a typical fish-eye imagethat contains an image circle as a whole in which each captured area isprojected, as illustrated in a fish-eye image A and a fish-eye image Bin FIG. 4A.

A plurality of fish-eye images of each frame captured by the pluralityof image sensors 130 is subjected to distortion correction and synthesisprocessing to form an omnidirectional image (spherical image) for eachframe. In the synthesis processing, an omnidirectional image, whichconstitutes a complementary hemispherical portion, is generated fromeach planar fish-eye image. Then, the two omnidirectional imagesincluding the respective hemispherical portions are joined together bymatching the overlapping areas of the hemispherical portions, and theomnidirectional images are synthesized to generate a full spherical(omnidirectional) image including the whole sphere.

FIG. 4B is an illustration of a plane data structure of the image dataof an omnidirectional image used in the omnidirectional moving-imagesystem according to an embodiment of the present disclosure. FIG. 4C isan illustration of a spherical data structure of the image data of anomnidirectional image used in the omnidirectional moving-image systemaccording to an embodiment of the present disclosure. As illustrated inFIG. 4B, the image data of the omnidirectional image is expressed as anarray of pixel values where the vertical angle φ corresponding to theangle with reference to a certain axis and the horizontal angle θcorresponding to the angle of rotation around the axis are thecoordinates. The vertical angle φ ranges from 0° to 180° (alternativelyfrom −90° to +90°), and the horizontal angle θ ranges from 0° to 360°(alternatively from −180° to +180°).

As illustrated in FIG. 4C, the respective coordinate values (θ, φ) ofthe omnidirectional format (the spherical data structure of the imagedata) are associated with the points on the sphere that represents alldirections from the photographing location. Thus, all directions areassociated with the points on the omnidirectional images. The planecoordinates of the fish-eye image captured by a fish-eye lens areassociated with the coordinates on the sphere of the omnidirectionalimage, which are included in a predetermined transformation table. Thetransformation table includes data prepared in advance by, for example,a manufacturer in accordance with a predetermined projection model basedon design data of each lens optical system. The data of thetransformation table is used for transforming a fish-eye image into anomnidirectional image.

In the description above, the omnidirectional frame data 242 is assumedto constitute an omnidirectional moving image. However, no limitation isintended thereby. There is no need for each frame to be recorded on theformat of the synthesized omnidirectional image (synthesizedomnidirectional image format) illustrated in FIG. 4C. In someembodiments, the omnidirectional frame data 242 is in any other form aslong as the omnidirectional moving image can be constructed forreproduction.

For example, assuming that fish-eye images for each frame are subjectedto distortion correction and synthesis processing to generate anomnidirectional image to be reproduced, moving-image data of twofish-eye images directly captured by the image sensors 130A and 130B(the respective pieces of moving-image data corresponding to thefish-eye image A and the fish-eye image B in FIG. 4A) is recorded as theomnidirectional frame data 242. Alternatively, moving-image data of acemented image formed by joining together the two fish-eye images A andB (that is moving-image data of one image formed by joining together thefish-eye image A and the fish-eye image B arranged side by side) may berecorded as the omnidirectional frame data 242. In the followingdescription, the omnidirectional frame data 242 is assumed to containimages of each frame in the distortion-corrected and synthesizedomnidirectional image format, for convenience of description.

Further, the omnidirectional frame data 242 is not limited tomoving-image data, and may be recorded in any form as long as a movingimage can be reproduced. For example, the omnidirectional frame data 242is recorded as moving-image data in which a plurality of frames arecompressed in a certain codec such as H.264/MPEG-4 advanced video coding(AVC) or H.265/High Efficiency Video Coding (HEVC). Further, althoughJoint Photographic Experts Group (JPEG) is a format that expresses amoving image as continuous still images, the moving-image data may berecorded as a continuous series of still images of a plurality offrames.

The following description is made with reference to FIG. 3. The zenithcorrection data 244 is time-series data of the inclination angle of theomnidirectional camera 110 with respect to the reference direction ofthe omnidirectional camera 110 during the time from a start to an end ofa shooting operation. The time series data is recorded in associationwith each frame of an omnidirectional image. The reference direction ofthe omnidirectional camera 110 is typically the direction of gravity inwhich acceleration of gravity is applied. The inclination angle withrespect to the direction of gravity is generated based on the signalmeasured by the acceleration sensor included in the sensor 136. Theinclination angle is typically configured as a vector of accelerationdimension values. Since the acceleration sensor does not distinguishbetween gravity and inertial force, preferably the inclination angleobtained from the acceleration sensor of the sensor 136 may be correctedbased on the signal measured by the angular velocity sensor.

In the embodiments of the present disclosure, the inclination anglescorrespond to the frames on a one-by-one basis, and the inclinationangles and the frames are recorded synchronously. However, the rate ofthe inclination angle to be recorded may not be the same as the framerate. When the rate of the inclination angle is not the same as theframe rate, an inclination angle corresponding to the frame on aone-by-one basis is obtained by performing resampling at the frame rate.

The angular velocity data 246 is time-series data of the angularvelocities generated around three axes of the angular velocity sensor ofthe omnidirectional camera 110 during the time from a start to an end ofa shooting operation, measured by the angular velocity sensor of thesensor 136. The angular velocities may not be recorded in associationwith the frames. Typically, angular velocities faster than the framerates are recorded in the angular velocity data 246. In this case, byusing the time stamp, the relations between the frames and theinclination angles are obtained at a later time. Alternatively, theangular velocities may be recorded in association with the frames of theomnidirectional moving images, respectively.

The omnidirectional moving-image correction unit 220 reads out theomnidirectional moving-image data 240 stored in the storage unit 214,and applies zenith correction and rotation correction to each frame ofthe omnidirectional frame data 242 (in other words, the omnidirectionalmoving-image correction unit 220 rotates an image of each frame), thusoutputting the corrected omnidirectional moving image data to anotherunit.

The following describes the zenith correction and the rotationcorrection with reference to FIGS. 5 and 6. FIG. 5 is an illustration ofthe zenith correction and the rotation correction applied to anomnidirectional image according to an embodiment of the presentdisclosure. FIGS. 6A and 6B are illustrations of an omnidirectionalimage obtained by performing the zenith correction and the rotationcorrection according to an embodiment of the present disclosure. FIG. 6Ais an illustration of a frame of a moving image before the zenithcorrection is made, and FIG. 6B is an illustration of a frame of themoving image after the zenith correction is made.

As described above, the image data of an omnidirectional image format isexpressed as an array of pixel values where the vertical angle φcorresponding to the angle with reference to a certain axis z0 and thehorizontal angle θ corresponding to the angle of rotation around theaxis z0 are the coordinates. If no correction is made, the certain axisz0 is defined with reference to the omnidirectional camera 110. Forexample, the axis z0 is defined as the central axis z0, which definesthe horizontal angle θ and the vertical angle φ, passing through thecenter of the housing 14 from the bottom to the top where the top is theimaging body 12 side and the bottom is the opposite side of theomnidirectional camera 110 in FIG. 1. Further, for example, thehorizontal angle θ of an omnidirectional image is defined such that thedirection of the optical axis of the optical element of one of theimage-forming optical system 20A and 20B is positioned at the center ofthe corresponding image sensor 22 at the horizontal angle θ.

The zenith correction (correction in the direction of roll and thedirection of pitch) is correction processing that corrects theomnidirectional images (FIG. 6A) captured with the central axis z0actually inclined with respect to the reference direction Z (thedirection of gravity) as illustrated in the left illustration of FIG. 5,to an omnidirectional image (FIG. 6B) captured with the central axis z0aligned with the reference direction Z as illustrated in the rightillustration of FIG. 5. The rotation correction is a correction(correction in Yaw direction) that reduces rotational distortion aroundthe reference direction Z (in the horizontal angle θ direction in FIGS.6A and 6B) in the omnidirectional image to which the zenith correctionhas been made to have the central axis z0 aligned with the referencedirection Z.

The operation of the omnidirectional moving-image correction unit 220 isdescribed below in detailed with reference to FIG. 3. More specifically,the omnidirectional moving-image correction unit 220 includes a camerafront calculation unit 222 (a first calculation unit), a correctionangle calculation unit 224 (a second calculation unit), a correctionangle adjusting unit 226 (a third calculation unit), and an imagerotating unit 228.

Based on the zenith correction data 244 and the angular velocity data246, the camera front calculation unit 222 calculates time-series valuesof the front direction (the direction of the optical axis of one of theimage-forming optical systems 20A and 20B) of the omnidirectional camera110. More specifically, the camera front calculation unit 222 firstcalculates the front direction V (0) (first front direction) of theomnidirectional camera 110 in capturing the leading frame (at the firsttime), based on a value T (0) of the inclination angle of the leadingframe included in the zenith correction data 244. After obtaining thefront direction V (0) for the leading frame, the camera frontcalculation unit 222 calculates the time-series values of the frontdirection V(n) of the omnidirectional camera 110 over a plurality oftimes corresponding to the respective frames based on the angularvelocity Gyro (n) of the angular velocity data 246, starting from theinitial value V (0). The front direction V(n) is obtained by integratinginfinite small rotation corresponding to the angular velocity of eachaxis, starting from the front direction V (0) for the leading frame. Thecalculated time-series values of the front direction V(n) aretransmitted to the correction angle calculation unit 224.

Based on the inclination angle data T(m) of the zenith correction data244, the correction angle calculation unit 224 calculates the correctionangle Angle (n) of rotation around the reference direction Z incapturing each frame, from the front direction V(n) of theomnidirectional camera 110 in capturing each corresponding frame. Thecalculated correction angle Angle (n) is transmitted to the correctionangle adjusting unit 226.

The correction angle adjusting unit 226 adjusts the calculatedcorrection angle Angle (n) for each frame to achieve a successfulnatural correction operation. The obtained (calculated) correction angleAngle (n) is typically within a prescribed domain (for example, from−180 degrees to +180 degrees). Accordingly, the correction angle Angle(n) might change from the vicinity of +180 degrees to the vicinity of−180 degrees (or in the opposite direction from the vicinity of −180degrees to the vicinity of +180 degrees) when viewed as a time-seriesvalues. In view of such a situation, the correction angle adjusting unit226 detects a change of the correction angle Angle (n) from the vicinityof the upper limit to the vicinity of the lower limit (from the vicinityof the lower limit to the vicinity of the upper limit) of the domainbased on the amount of change in the correction angle Angle (n) betweencontinuously obtained correction angles. In other words, the correctionangle adjusting unit 226 detects a change of the correction angle Angle(n) from an extreme to another extreme of the domain of the correctionangle. Based on such a detection, the correction angle adjusting unit226 adds or subtracts an adjustment value to or from the correctionangle Angle (n) for each frame. With such a configuration, the domain ofthe correction angle is expanded and the continuity of the time-seriescorrection angles. Further, the correction angle adjusting unit 226 (afiltering unit) performs a filtering process on the correction angleswithin the expanded domain to allow for passage of only frequencycomponent within a predetermined frequency range, preferably a highfrequency component. The corrected angle Angle Out (n) on which thefiltering process has been performed is transmitted to the imagerotating unit 228.

Based on the zenith correction data 244, the image rotating unit 228performs the zenith correction on an omnidirectional image for each ofthe plurality of frames of the omnidirectional frame data 242. At thesame time, the image rotating unit 228 according to the presentembodiment rotates the images based on the calculated correction anglesAngle Out (n) for the plurality of frames, so as to reduce therotational distortion around the reference direction Z within the rangeof the predetermined frequency.

When the image of each frame in the omnidirectional frame data 242 is animage of the synthesized omnidirectional image format, the imagerotating unit 228 transforms the rotational coordinates of theomnidirectional image for each frame. When each frame in theomnidirectional frame data 242 includes a plurality of fish-eye imagesbefore the synthesis, the image rotating unit 228 applies the zenithcorrection and the rotation correction to each frame, and alsotransforms the plurality of fish-eye images into an omnidirectionalimage.

The receiving unit 230 receives various requests from the informationterminal 150 communicable with the omnidirectional camera 110. Uponreceipt of a request to correct an omnidirectional moving image from theinformation terminal 150 (for example, a request to output a correctedomnidirectional moving image for reproduction), the receiving unit 230transfers the request to the omnidirectional moving-image correctionunit 220. In response to the request to correct the omnidirectionalmoving image, the omnidirectional moving-image correction unit 220starts performing the zenith correction and the rotation correction onthe designated omnidirectional moving-image data 240 to output thecorrected image data. The image output unit 232 outputs the image dataof the omnidirectional moving image based on the plurality of frames onwhich the zenith correction and the rotation correction has beenperformed by the omnidirectional moving-image correction unit 220, tothe information terminal 150 of the request source. In this case, theimage data may be encoded in a predetermined moving-image compressionformat based on the corrected omnidirectional images for the respectiveplurality of frames, and output as final moving-image data or as aseries of still images.

The information terminal 150 is a terminal device such as a smartphone,a tablet computer, and a personal computer in which an application forcommunicating with the omnidirectional camera 110 to view and reproduceomnidirectional images is installed. The information terminal 150receives various instructions from the operator via the application andissues various requests to the omnidirectional camera 110. In responseto accepting an instruction of the operator to correct theomnidirectional moving image designated by the operator (for example, aninstruction to reproduce a moving image while correcting), theinstruction receiving unit 252 of the information terminal 150 issues,to the omnidirectional camera 110, a request to output moving-image dataof a certain corrected omnidirectional moving image. The display unit254 of the information terminal 150 displays an omnidirectional movingimage on the display device such as the display 162 of the informationterminal 150 based on the image data of the omnidirectional moving imageoutput from the omnidirectional camera 110.

Note that the information terminal 150 displays any desired type ofimage based on the corrected image data. For example, theomnidirectional image as is may be displayed on the display device.Alternatively, the omnidirectional image may be pasted on the sphericalobject, and images captured when the spherical object is observed with acamera of a predetermined viewing angle from a predetermined positionare displayed as frames to display a moving image.

In the present embodiment, the zenith correction and rotation correctionprocesses are actually performed using the resources of theomnidirectional camera 110, not the information terminal 150, and thecorrection result is output to the information terminal 150 to display acorrected image. With this configuration, regardless of the processingperformance of the information terminal 150, it is possible to stablyreproduce moving-images reproduction while applying zenith correctionand rotation correction to images.

In the embodiments of the present disclosure, the cases where the imagedata of the omnidirectional moving image is transmitted to theinformation terminal 150 are described. However, no limitation isintended thereby. When the omnidirectional camera 110 includes a displaydevice, the display device of the omnidirectional camera 110 may displayan omnidirectional moving image. Alternatively, the image data of theomnidirectional moving image may be stored as another file.

The rotation correction is described below in more detail according toan embodiment of the present disclosure with reference to FIGS. 7 and 8.FIG. 7 is a flowchart of an omnidirectional moving-image correctionprocessing performed by the omnidirectional camera 110 that constitutesthe omnidirectional moving-image system 1000 according to an embodimentof the present disclosure. FIGS. 8A and 8B are illustrations forexplaining a method of calculating a front vector for the rotationcorrection according to an embodiment of the present disclosure.

In the processing illustrated in FIG. 7, the omnidirectional camera 110starts the processing in response to a receipt of the request to outputmoving-image data of a corrected certain omnidirectional moving imagefrom the information terminal 150 that externally communicates with theomnidirectional camera 110.

In step S101, the omnidirectional camera 110 reads the omnidirectionalmoving-image data 240 designated by the received request. In this case,the omnidirectional moving-image data 240 read by the omnidirectionalcamera 110 includes the omnidirectional frame data 242 that constitutesthe omnidirectional moving-image data 240 and metadata including thezenith correction data 244 and the angular velocity data 246.

In step S102, the omnidirectional camera 110 calculates the frontdirection V (0) of the omnidirectional camera 110 for the leading frame(in capturing the leading (first) frame) based on the inclination anglevector T (0) of the leading frame in the zenith correction data 244.First, a quaternion Q (0) of the inclination angle vector T (0) of theleading frame and the direction of gravity vector G=(0, 0, 1) aregenerated. The inclination angle vector T (0) represents the rotationaldisplacement between the global coordinate system for the leading(first) frame and the local coordinate system of the omnidirectionalcamera 110. The rotational axis A (0) and the rotation angle θ(0) of thequaternion Q (0) are expressed by the following formulas (1) and (2).Then, the quaternion Q (0) represents the rotation of the rotation angleθ(0) about the rotation axis A (0) as illustrated in FIG. 8A.

$\begin{matrix}{{A(0)} = \frac{G \times {T(0)}}{{G \times {T(0)}}}} & (1) \\{{\theta (0)} = {{acos}\left( \frac{G \cdot {T(0)}}{{G}{{T(0)}}} \right)}} & (2)\end{matrix}$

As illustrated in FIG. 8A, the global front vector (0, 1, 0) is rotatedwith the obtained quaternion Q (0) to obtain the initial value V (0) ofthe front direction vector of the omnidirectional camera 110. In thiscase, assuming that the central axis z0 of the omnidirectional camera110 is aligned with the Z axis (direction of gravity) of the globalcoordinate system and the front direction (the optical axis of oneoptical system) of the omnidirectional camera 110 is aligned with Y axisof the global coordinate system, the initial value V (0) of the frontdirection vector that indicates the front of the omnidirectional camera110 in an inclined state.

In step S103, the omnidirectional camera 110 calculates time-seriesvalues of the front direction vector V(n) of the omnidirectional camera110 over a plurality of time points corresponding to the respectiveframes, based on the angular velocity data 246. The front directionvector V(n) at each time point is obtained by calculating the timedifference between sampling processes in the angular velocity data 246and integrating infinite small rotation corresponding to the angularvelocity Gyro (n) about three axes for each sampling. Although samplingintervals might vary due to the load on the omnidirectional camera 110,the time difference between sampling can be suitably obtained from thetime stamp. In step S103, as illustrated in FIG. 8B, the time-seriesvalues of the front direction vector V(n) are calculated, starting fromthe front direction vector V (0) with which the leading (first) frame iscaptured.

In the loop of steps S104 to S110, the zenith correction and rotationcorrection are applied for each frame while calculating the correctionangle for the rotation correction.

In step S105, the omnidirectional camera 110 searches the camera frontdirection vector V(n) corresponding to the frame of interest andcalculates the correction angle for the rotation correction based on thezenith correction data 244. Typically, the angular velocity data 246 hasa sampling rate different from the frame rate, in which a frontdirection vector V close to the time stamp and corresponding to a frameof interest of the zenith correction data 244 is searched.

In this case, it is assumed that the front direction vector V(n)corresponds to the m-th frame. First, quaternion Q(m) is obtained fromthe inclination angle vector T(m) of the m-th frame. The rotation axisA(m) and the rotation angle θ(m) of the quaternion Q(m) are expressed bythe following formulas (3) and (4) same as in the formulas (1) and (2).

$\begin{matrix}{{A(m)} = \frac{G \times {T(m)}}{{G \times {T(m)}}}} & (3) \\{{\theta (0)} = {{acos}\left( \frac{G \cdot {T(m)}}{{G}{{T(m)}}} \right)}} & (4)\end{matrix}$

Then, as illustrated in FIG. 8B, the front direction vector V(n)corresponding to the m-th frame is rotated with the quaternion Q (m) toobtain the front direction vector V′(n). As a result, the direction inwhich the front face vector V(n) of the camera faces after zenithcorrection is obtained. Then, based on the obtained vector V′(n), thecorrection angle Angle(n) is obtained within the XY plane of the globalcoordinate, using the following formula.

Angle(n)=a tan 2(V,x(n),V,y(n)  (5)

In this case, the function a tan 2 (x coordinate and y coordinate) inthe formula (5) above is a function that returns the arc tangent in therange of −180 degrees to 180 degrees (−π to π).

In step S106, as a pre-process of the high-pass filter, theomnidirectional camera 110 adjusts the correction angle obtained in stepS105 to maintain the continuity of the correction angles, and thuscalculates a correction angle AngleIn (n) after the adjustment.

In the embodiments described above, the correction angle Angle (n) isobtained as a value within the range of −180° to +180° by the arctangent. In such a case, the correction angle Angle (n) may change fromthe vicinity of +180 degrees to the vicinity of −180 degrees (or in theopposite direction from the vicinity of −180 degrees to the vicinity of+180 degrees) when viewed as a time-series values. In step S106, asindicated by the following pseudocode, the omnidirectional camera 110(an addition-subtraction unit) detects a change of the correction angleAngle (n) from the vicinity of the upper limit to the vicinity of thelower limit (from the vicinity of the lower limit to the vicinity of theupper limit) of the domain based on the amount of change in thecorrection angle Angle (n) between continuously obtained correctionangles. In other words, a change of the correction angle Angle (n) froman extreme to another extreme of the domain is detected. Based on such adetection, an adjustment value is added to or subtracted by theaddition-subtraction unit from the correction angle Angle (n) for eachframe. In the following pseudocodes, the “threshold” is a threshold forthe amount of change of the correction angle Angle (n) to detect thechange of the correction angle from the vicinity of the upper limit tothe vicinity of the lower limit (from the vicinity of the lower limit tothe vicinity of the upper limit) of the domain. The last code “Angle(0)”in the following formula is the initial value of the correction angleobtained by the above formula (5). Based on the correction angleAngle(0), the correction is performed. In the following pseudocodes,“(n⁻¹)” corresponds to the frame immediately before the frame to which ncorresponds.

  if (angle(n) − angle(n⁻¹ > threshold) {  AdjustAngle(n) =AdjustAngle(n⁻¹) − 360 } else if (angle(n) − angle(n⁻¹) < −threshold){ AdjustAngle(n) = AdjustAngle(n⁻¹) + 360 } else{  AdjustAngle(n) =AdjustAngle(n⁻¹) } AngleIn(n) = Angle(n) + AdjustAngle(n) − Angle(0)

In step S107, the omnidirectional camera 110 performs a high-passfiltering process on the adjusted correction angle AngleIn (n) for theframe of interest to allow for passage of the high-frequency component.Thus, the filtered correction angle AngleOut (n) after the filteringprocess is obtained. The filtered correction angle AngleOut (n) isobtained by the following formula. The symbol “P” in the followingformula is a sampling period, and the symbol “Hc” is a cutoff frequency.In the following formula, (n₊₁) corresponds to the frame immediatelyafter the frame corresponding to “n”.

${{AngleOut}(n)} = {{{AngleIn}(n)} - {\sum\frac{{Angle}(n)}{t}}}$∑Angle(n₊₁) = ∑Angle(n) + AngleOut(n) t = P/(2π * Hc)

In step S108, in addition to the zenith correction, the omnidirectionalcamera 110 performs the rotation correction on the frame of interest inthe omnidirectional frame data 242, based on the calculated correctionangle and a corresponding inclination vector T (m) in the zenithcorrection data 244, so as to reduce the rotational distortion aroundthe reference direction due to the high-frequency component. In stepS109, the omnidirectional camera 110 outputs a frame of theomnidirectional image that has been subjected to the rotationaltransformation.

When the processes of steps S105 to S109 are performed on all the framesincluded in the omnidirectional moving-image data 240, the loop of stepsS104 to S110 ends and the correction processing ends.

When the omnidirectional camera 110 is properly placed in the verticaldirection (the omnidirectional camera 110 is not inclined with referenceto the reference direction (direction of gravity)), it is possible toshoot an omnidirectional image such that the zenith is aligned with thehorizontal line, which is recognized by the shooter, as illustrated inFIG. 6B. In general, however, it is difficult to capture an image withthe position and posture secured accurately while holding the camerawith a hand. If an image is captured with the camera body inclined, thezeniths of the images are misaligned and the horizontal line isdistorted unless the zenith correction is performed, as illustrated inFIG. 6A. If the camera body rotates due to, for example, camera shake,the image blurs along the horizontal line unless the rotation correctionis performed.

However, in the embodiments according to the present disclosure, theimages for all the frames are subjected to the transformation process(the correction processing) such that the reference direction Z such asthe direction of gravity is aligned with the central axis z0 of theomnidirectional camera 110 and such that the rotational distortion of animage around the reference direction Z is reduced. With such aconfiguration, the viewer can view the corrected omnidirectional movingimage without feeling uncomfortable.

As described above, the configuration according to the above-describedembodiments can provide an image processing apparatus capable ofeffectively correcting rotational blur of an image around a prescribedreference direction, an image processing system incorporating the imageprocessing apparatus, and carrier means.

Particularly, the rotation correction is performed using angularvelocity data stored as metadata in shooting, which can reduce the loadduring the shooting. Further, it is possible to decide to not performthe rotation correction or to perform the rotation correction so as toprevent only a very small component, at the time of reproduction.

In the above-described embodiments, the omnidirectional camera 110 aredescribed as an example of an image processing apparatus. With theconfiguration that performs the substantive processing of the zenithcorrection and the rotation correction using the resources of theomnidirectional camera 110, a moving image can be stably reproducedwhile applying zenith correction and rotation correction to images,regardless of the processing performance of the apparatus that serves toreproduce the moving image. However, the image processing apparatus maynot be an image-capturing device such as the omnidirectional camera 110.In some other embodiments, an information processing apparatus such as apersonal computer different from the omnidirectional camera 110 may beused as the image processing apparatus.

In the above-described embodiment, the cases where two partial imagescaptured by the lens optical systems each having an angle of viewgreater than 180 degrees are superimposed and synthesized are described.However, no limitation is intended thereby. In some other embodiments,three or more partial images captured by one re more lens opticalsystems may be superimposed on each other and synthesized. Further, theconfiguration according to the above-described embodiments are appliedto an imaging system equipped with a fish-eye lens. Alternatively, theconfiguration according to the embodiments of the present disclosure isapplicable in an omnidirectional moving-image imaging system equippedwith a super-wide-angle lens.

Further, in the above-described embodiments, the cases where theomnidirectional camera 110 is separate from the information terminal 150in the omnidirectional moving-image system 1000. However, no limitationis intended thereby. In some embodiments, the omnidirectional camera 110may be combined with the information terminal 150 in the omnidirectionalmoving-image system 1000.

Further, in the above-described embodiments, the omnidirectional camera110 is not inclined when the direction of gravity is aligned with thecentral axis of the omnidirectional camera 110. However, no limitationis intended thereby. Instead of the direction of gravity, for example,the horizontal direction or another desired direction may be set as areference direction, and the inclination of the image may be correctedbased on the inclination of a prescribed object, such as theomnidirectional camera 110 or the image sensor 130A or 130B, withreference to the reference direction.

The functional blocks as described above are implemented by acomputer-executable program written by programming languages such as anassembler language, C, and object-oriented programming languages such asC++, C #, and Java (registered trademark). The program may bedistributed via a telecommunication line as being stored in acomputer-readable storage medium such as a ROM, an electrically erasableand programmable read only memory (EEPROM), an electrically programmableread only memory (EPROM), a flash memory, a flexible disk, a compactdisc read only memory (CD-ROM), a compact disc rewritable (CD-RW), adigital versatile disk (DVD)-ROM, a DVD-RAM, a DVD-RW, a Blu-ray disc, asecure digital (SD) card, and a magneto-optical disc (MO). All or someof the functional units described above can be implemented, for example,on a programmable device such as a field programmable gate array (FPGA),or as an application specific integrated circuit (ASIC). To implementsuch functional units on the programmable device, circuit configurationdata (bit stream data) to be downloaded to the programmable device canbe distributed using a recording medium that stores data written in, forexample, a hardware description language (HDL), Very High SpeedIntegrated Circuit Hardware Description Language (VHDL), or Verilog HDL.

Numerous additional modifications and variations are possible in lightof the above teachings. It is therefore to be understood that, withinthe scope of the appended claims, the disclosure of this patentspecification may be practiced otherwise than as specifically describedherein. The present invention can be implemented in any convenient form,for example using dedicated hardware, or a mixture of dedicated hardwareand software. The present invention may be implemented as computersoftware implemented by one or more networked processing apparatuses.The processing apparatuses can compromise any suitably programmedapparatuses such as a general purpose computer, personal digitalassistant, mobile telephone (such as a WAP or 3G-compliant phone) and soon. Since the present invention can be implemented as software, each andevery aspect of the present invention thus en-compasses computersoftware implementable on a programmable device. The computer softwarecan be provided to the programmable device using any conventionalcarrier medium (carrier means). The carrier medium can compromise atransient carrier medium such as an electrical, optical, microwave,acoustic or radio frequency signal carrying the computer code. Anexample of such a transient medium is a TCP/IP signal carrying computercode over an IP network, such as the Internet. The carrier medium canalso comprise a storage medium for storing processor readable code suchas a floppy disk, hard disk, CD ROM, magnetic tape device or solid statememory device. Although the embodiments of the present disclosure havebeen described above, the present disclosure is not limited to theembodiments described above, but a variety of modifications cannaturally be made within the scope of the present disclosure. Numerousadditional modifications and variations are possible in light of theabove teachings. It is therefore to be understood that within the scopeof the appended claims, the embodiments may be practiced otherwise thanas specifically described herein. For example, elements and/or featuresof different illustrative embodiment s may be combined with each otherand/or substituted for each other within the scope of this disclosureand appended claims.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), DSP (digital signal processor), FPGA (fieldprogrammable gate array) and conventional circuit components arranged toperform the recited functions.

The illustrated apparatuses are only illustrative of one of severalcomputing environments for implementing the embodiments disclosedherein. For example, in some embodiments, the image-capturing deviceincludes a plurality of computing devices, e.g., a server cluster, thatare configured to communicate with each other over any type ofcommunications link, including a network, a shared memory, etc. tocollectively perform the processes disclosed herein. Similarly, theimage processing apparatus can include a plurality of computing devicesthat are configured to communicate with each other.

Moreover, the image-capturing device and the image processing apparatuscan be configured to share the processing steps disclosed, e.g., in FIG.7, in various combinations. For example, the processes performed by thecorrection unit can be performed by the image processing apparatus.Similarly, the functionality of the correction unit can be performed bythe image processing apparatus. Further, the illustrated elements of theimage-capturing device and the image processing apparatus can becombined into a single server apparatus, or divided between a pluralityof machines in combinations other than that shown in any of theabove-described figures.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application No. 2017-242107, filed onDec. 18, 2017 in the Japan Patent Office, the entire disclosure of whichis hereby incorporated by reference herein.

REFERENCE SIGNS LIST

-   -   12 Imaging body    -   14 Housing    -   18 Shutter button    -   20 Image-forming optical system    -   22, 130 Image sensor    -   110, 130 Omnidirectional camera    -   112, 152 CPU    -   114 ROM    -   116 Image processing block    -   118 Moving-image compression block    -   120, 126 Interface    -   122 Storage interface    -   124 Sensor interface    -   126 USB interface    -   128 Serial block    -   132 DRAM    -   134 External storage    -   136 Sensor    -   138, 166 USB connector    -   150 Information terminal    -   154 RAM    -   156 Internal storage    -   158 Input device    -   160 Storage    -   162 Display    -   164 Wireless NIC    -   210, 250 Functional block    -   212 Omnidirectional moving-image capturing unit    -   214 Storage unit    -   220 Omnidirectional moving-image correction unit    -   222 Camera front calculation unit    -   224 Correction angle calculation unit    -   226 Correction angle adjusting unit    -   228 Image rotating unit    -   230 Receiving unit    -   232 Image output unit    -   240 Omnidirectional moving-image data    -   242 Omnidirectional frame data    -   244 Zenith correction data    -   246 Angular velocity data    -   252 Instruction receiving unit    -   254 Display unit

1. An image processing apparatus, comprising: memory to storemoving-image data including a plurality of frames captured by animage-capturing device communicable with the image processing apparatus,time-series data of an inclination angle of the image-capturing devicewith reference to a reference direction, and time-series data of anangle velocity of the image-capturing device; and circuitry configuredto, based on the time-series data of the angle velocity, rotate an imageof each of the plurality of frames of the moving-image data to reduce arotational distortion around the reference direction within a prescribedfrequency range, and output image data of the rotated image of each ofthe plurality of frames to an external device communicable with theimage processing apparatus.
 2. The image processing apparatus accordingto claim 1, wherein, in response to a request, from the external device,to output a moving image of the rotated image of each of the pluralityof frames, the circuitry is further configured to start rotating animage of each of the plurality of frames of the moving-image data tooutput image data of the rotated image of each of the plurality offrames.
 3. The image processing apparatus according to claim 1, whereinthe image to be rotated by the circuitry is an omnidirectional image,and wherein, in rotating the image, the circuitry is further configuredto perform a zenith correction on the omnidirectional image of each ofthe plurality of frames of the moving-image data, based on thetime-series data of the inclination angle.
 4. The image processingapparatus according to claim 1, wherein the circuitry is furtherconfigured to: calculate a first front direction of the image-capturingdevice at a first time based on a value of the time-series data of theinclination angle at the first time; calculate time-series values of afront direction of the image-capturing device over a plurality of timescorresponding to the plurality of frames based on the time-series dataof the angle velocity; and calculate a correction angle at each of theplurality of times in rotating the image from the front direction ateach of the plurality of times, based on the time-series data of theinclination angle.
 5. The image processing apparatus according to claim4, further comprising a filter configured to perform a filtering processon the calculated correction angle at each of the plurality of times toallow for passage of a high-frequency component within the prescribedfrequency range.
 6. The image processing apparatus according to claim 4,wherein the circuitry is further configured to detect a change of thecorrection angle from an extreme to another extreme of a domain of thecorrection angle and to add or subtract an adjustment value to or fromthe correction angle.
 7. The image processing apparatus according toclaim 4, wherein the circuitry is further configured to obtain the frontdirection at each of the plurality of times by integrating infinitesmall rotation corresponding to the angular velocity at each axis of thetime-series data of the angular velocity, starting from the first frontdirection at the first time.
 8. An image processing system, comprising:memory to store moving-image data including a plurality of framescaptured by an image-capturing device communicable with the imageprocessing apparatus, time-series data of an inclination angle withreference to a reference direction of the image-capturing device, andtime-series data of an angle velocity of the image-capturing device; andcircuitry configured to, based on the time-series data of the anglevelocity, rotate an image of each of the plurality of frames of themoving-image data to reduce a rotational distortion around the referencedirection within a prescribed frequency range, and output image data ofthe rotated image of each of the plurality of frames to an externaldevice communicable with the image processing apparatus.
 9. The imageprocessing system according to claim 8, further comprising: theimage-capturing device including the memory, and the circuitry; and aninformation processing apparatus communicable with the image-capturingdevice, the information processing apparatus comprising: a receiver toreceive a request to output the rotated moving-image; a display todisplay image data output from the image-capturing device; wherein, inresponse to a request, from the external device, to output a movingimage of the rotated image of each of the plurality of frames, thecircuitry is further configured to start rotating an image of each ofthe plurality of frames of the moving-image data to output image data ofthe rotated image of each of the plurality of frames.
 10. A method ofprocessing an image, the method comprising: storing moving-image dataincluding a plurality of frames captured by an image-capturing devicecommunicable with the image processing apparatus, time-series data of aninclination angle with reference to a reference direction of theimage-capturing device, and time-series data of an angle velocity of theimage-capturing device; based on the time-series data of the anglevelocity, rotating an image of each of the plurality of frames of themoving-image data to reduce a rotational distortion around the referencedirection within a prescribed frequency range; and outputting image dataof the rotated image of each of the plurality of frames to an externaldevice communicable with the image processing apparatus.
 11. (canceled)